-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-32824][Core] Improve the error message when the user forgets the .amount in a resource config #29685
Conversation
.amount after a resource like spark.executor.resource.gpu
Test build #128421 has finished for PR 29685 at commit
|
Merged to master and branch-3.0. |
…he .amount in a resource config ### What changes were proposed in this pull request? If the user forgets to specify .amount on a resource config like spark.executor.resource.gpu, the error message thrown is very confusing: ``` ERROR SparkContext: Error initializing SparkContext.java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1967) at ``` This makes it so we have a readable error thrown ### Why are the changes needed? confusing error for users ### Does this PR introduce _any_ user-facing change? just error message ### How was this patch tested? Tested manually on standalone cluster Closes #29685 from tgravescs/SPARK-32824. Authored-by: Thomas Graves <tgraves@nvidia.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit e8634d8) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
@tgravescs, it's just nothing but my curiosity, why don't you add your NVIDIA email into your GitHub account so the commit links your GitHub account :-)? |
no particular reason, thanks for pointing it out. |
Yeah, there was no conflict and tests passed (e86d90b). |
great thanks |
…he .amount in a resource config ### What changes were proposed in this pull request? If the user forgets to specify .amount on a resource config like spark.executor.resource.gpu, the error message thrown is very confusing: ``` ERROR SparkContext: Error initializing SparkContext.java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1967) at ``` This makes it so we have a readable error thrown ### Why are the changes needed? confusing error for users ### Does this PR introduce _any_ user-facing change? just error message ### How was this patch tested? Tested manually on standalone cluster Closes apache#29685 from tgravescs/SPARK-32824. Authored-by: Thomas Graves <tgraves@nvidia.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit e8634d8) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
What changes were proposed in this pull request?
If the user forgets to specify .amount on a resource config like spark.executor.resource.gpu, the error message thrown is very confusing:
This makes it so we have a readable error thrown
Why are the changes needed?
confusing error for users
Does this PR introduce any user-facing change?
just error message
How was this patch tested?
Tested manually on standalone cluster